Skip to content

feat: monitoring retention policy and ingest hardening (#540, #543)#739

Open
2witstudios wants to merge 6 commits intomasterfrom
ppg/monitoring-hardening
Open

feat: monitoring retention policy and ingest hardening (#540, #543)#739
2witstudios wants to merge 6 commits intomasterfrom
ppg/monitoring-hardening

Conversation

@2witstudios
Copy link
Owner

@2witstudios 2witstudios commented Feb 28, 2026 •
edited
Loading

Summary

  • Retention policies for monitoring tables that previously grew unbounded: api_metrics 90d, system_logs 30d, security_audit_log 365d — configurable via RETENTION_*_DAYS env vars
  • Ingest hardening with PII redaction (query params stripped from stored URLs), payload sanitization (truncation, clamping), and graceful degradation for missing MONITORING_INGEST_KEY
  • Integrated monitoring cleanup into existing runRetentionCleanup pipeline
  • Performance: Parallelized independent DB writes in monitoring ingest route (writeApiMetrics, systemLogs insert, writeError now run concurrently via Promise.all)
  • Added workflows to full-page route pattern in dashboard layout

Changes

  • packages/lib/src/compliance/retention/monitoring-retention.ts — new retention cleanup functions for api_metrics, system_logs, security_audit_log
  • packages/lib/src/compliance/retention/retention-engine.ts — integrated monitoring retention into unified cleanup pipeline
  • apps/web/src/lib/monitoring/ingest-sanitizer.ts — URL redaction, payload sanitization (truncation, clamping, query stripping)
  • apps/web/src/middleware/monitoring.ts — apply endpoint sanitization, remove query from payloads
  • apps/web/src/app/api/internal/monitoring/ingest/route.ts — apply sanitizer, parallelize DB writes, improve missing key handling
  • apps/web/src/app/dashboard/DashboardLayoutClient.tsx — add workflows to full-page route regex

Test plan

  • Retention config defaults (90d/30d/365d) verified
  • Env var override and invalid value fallback tested
  • Query params redacted from URLs
  • Payload fields truncated/clamped to safe limits
  • Missing MONITORING_INGEST_KEY warns instead of errors
  • 34 new unit tests passing (13 retention + 21 ingest sanitizer)
  • Existing monitoring tests unaffected
  • Lint and typecheck pass
  • Merged with latest master (resolved checkMCPPageScope mock issue)

2witstudios and others added 2 commits February 27, 2026 22:53
@2witstudios @claude
feat: add monitoring retention policies for api_metrics, system_logs,…
7ef1237 
… security_audit_log (#540)

Add time-based retention cleanup for monitoring tables that previously
grew unbounded. Defaults: api_metrics 90d, system_logs 30d,
security_audit_log 365d. Configurable via RETENTION_*_DAYS env vars.
Integrated into existing runRetentionCleanup pipeline.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@2witstudios @claude
feat: harden monitoring ingest with PII redaction and payload sanitiz…
8edc169 
…ation (#543)

Redact query parameters from URLs before storage in api_metrics and
system_logs. Sanitize ingest payloads (truncate error/stack/userAgent,
clamp duration, strip query field). Downgrade missing MONITORING_INGEST_KEY
from error to warning for graceful degradation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@chatgpt-codex-connector
Copy link

chatgpt-codex-connector bot commented Feb 28, 2026

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Feb 28, 2026 •
edited
Loading

Warning

Rate limit exceeded

@2witstudios has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 15 minutes and 46 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: c83372b5-c5f1-41d1-90c2-0c091346b187

📥 Commits

Reviewing files that changed from the base of the PR and between b704a06 and cf77d66.

📒 Files selected for processing (9)
  • apps/web/src/app/api/internal/monitoring/ingest/route.ts
  • apps/web/src/lib/monitoring/__tests__/ingest-sanitizer.test.ts
  • apps/web/src/lib/monitoring/ingest-sanitizer.ts
  • apps/web/src/middleware/monitoring.ts
  • packages/lib/package.json
  • packages/lib/src/compliance/retention/monitoring-retention.test.ts
  • packages/lib/src/compliance/retention/monitoring-retention.ts
  • packages/lib/src/compliance/retention/retention-engine.test.ts
  • packages/lib/src/compliance/retention/retention-engine.ts
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch ppg/monitoring-hardening

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

2witstudios and others added 4 commits February 28, 2026 08:10
@2witstudios @claude
fix: update retention-engine integration test for new monitoring tables
029a245 
The runRetentionCleanup function now returns 12 results (9 original +
3 monitoring tables) after integrating monitoring retention cleanup.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@2witstudios
fix: deduplicate IngestPayload type and hoist middleware computations
4640dd1 
- Export IngestPayload from ingest-sanitizer.ts and import in route.ts,
  eliminating the duplicate 25-line interface definition
- Hoist sanitizeEndpoint() and getRequestSize() before try/catch in
  monitoring middleware, removing redundant calls in the error handler
@2witstudios
Merge remote-tracking branch 'origin/master' into ppg/monitoring-hard…
e35e86a 
…ening
@2witstudios
refactor: parallelize independent DB writes in monitoring ingest
cf77d66 
The three DB operations (writeApiMetrics, insert systemLogs, writeError)
are independent and can run concurrently via Promise.all, reducing
per-request latency from sequential X+Y+Z to max(X,Y,Z).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@2witstudios